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ABSTRACT 



This paper presents the results of research into a technique 
for the measurement of effective teaching that avoids the problems inherent 
in student evaluations of teacher effectiveness. The method takes as little 
as 5 minutes to administer and so can be used in- course by the teacher to 
track his or her teaching effectiveness. It results in a single number that 
can be used at the end of a course as an administrative decision point 
measure. The initial research with 3 classes of teacher training students 
(n=12, n=12, and n=23) was done in an Australian university and is now being 
replicated in other universities. The measure operationally defines the three 
basic abilities of the Three Ability Framework (3AF) : technical skills, 
professional competence, and professional attitudes. The 3AF method assesses 
the degree of alignment between the changes the students expect in these 
three abilities and the changes toward which the teacher is working. 
Correlations between the scores of academic attainment, degree of 
teacher/student alignment, and course satisfaction indicated that when the 
students and their teacher were working towards the same proportion and 
amount of these three abilities, the students had higher academic attainment 
and greater course satisfaction. The four steps of the application of the 
method are outlined. (Contains 1 table and 28 references.) (SLD) 
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A Technique for Measuring Effective Teaching of Professional Courses 



The traditional method of measuring teacher effectivness is by using student opinioimaires. 
Their formats differ in (a) the questions that are asked, (b)whether they are open ended or forced 
choice and (c) whether they are anonymous or confidential. Whatever format is chosen puts 
some limitations on how the results should be used. Unfortunately, the same format must serve 
purposes as diverse as improving instruction, informing student choice and making tenure and 
promotion decisions. Accusations referring to popularity contests, grade inflation, techniques 
that bias results, student/teacher power play, inappropriate statistical analyses, infringement of 
professional choice, legal inadequacy for institutional decisions, etc. indicate that this traditional 
method is not fulfilling all of its intended purposes. 

This paper presents the results of research into a technique for the measurement of 
effective teaching that avoids these problems. It takes as little as 5 minutes to administer and so 
can be used in-course by the teacher to track his or her teaching effectiveness. It results in a 
single number that can be used at the end of a course as an administration decision point measure 
of effective teaching. 

Effective teaching is here defined as maximising student academic attainment, and 
teacher and student course satisfaction. The initial research with three classes of teacher training 
students (n=21, 12,23) was done in an Australian University and is now being replicated in other 
universities. The measurement method operationally defines three basic abilities. For in-course 
tracking of teaching, the method assesses the degree of alignment between the changes the students 
expect in these three abilities and the changes the teacher is working towards. For end-of-course 
measurement of teaching effectiveness the method assesses the alignment of the teacher’s 
intentions with the students’ perception. 

Correlations between scores of academic attainment (coursework and examinations), 
degree of teacher/student alignment, and course satisfaction (both student’s and teacher’s), indicate 
that when the students and their teacher are working towards the same proportion and amount of 
these three abilities then those students have a higher academic attainment and greater course 
satisfaction, and the teacher finds greater enjoyment in teaching those students. A sensitivity 
analysis of the same data also showed that student/teacher alignment on these three abilities 
significantly predicted academic achievement (p<0.001) and course satisfaction (p=0.002). 
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Measuring Effective Teaching 



Widespread problems in the uses of student evaluations of teaching 

The use of student opinionnaires for the assessment of faculty teaching is now widespread in the 
Europe and the USA (Crumbley, 1995; Husbands & Fosh, 1993; Seldin, 1984, 1993; Yunker & Sterner, 
1988). Within each institution the results from the same set of forms are used to serve multiple purposes; 
for feedback to lecturers for improving their teaching, for publication to the student body for course 
registration decisions, for administrative course evaluation, and for promotion and tenure decisions 
(Kolevzon, 1981, Avi-Itzhak & Lya, 1986). However, it seems that student evaluations are failing to 
satisfactorily serve any of these purposes (Greenwald, 1997; Greenwald & Gillmore, 1997; Howard & 
Maxwell, 1982; Marsh & Dunkin, 1997; Marsh & Roche, 1997, 1998). 



There are widespread doubts about the validity of these forms; if they really do assess the quality of 
teaching, and whether students are qualified assessors (Chacko, 1983; Dowell & Neal, 1982; Powell, 
1977; Snyder & Clair, 1976; Vasta & Sarmiento, 1979; Worthington & Wong, 1979). A major problem is 
the influence that their use for faculty employment decisions is having on the conduct of university teaching 
and course assessment. “The higher education rhetoric is almost universal in stating that the primary 
purpose of faculty evaluation is to help faculty improve their performance. However, an examination of 
the systems used indicates that the primary purpose is almost always to make personnel decisions. That is, 
to make decisions for retention, promotion, tenure, and salary increases.” (Cashin, 1996). Students rating 
reflects the philosophy of customer service satisfaction - how much the lecturer met their requirements of 
being able to pass the course without an inconvenient workload and tedious attendance. This invites negative 
evaluations from students in the lower attainment and attendance range of the class. The most expedient 
career solution is to teach entertainingly at lower standards and use the time saved for career building 
research (Greenwald, 1996; Greenwald & Gillmore, 1966; DuCette & Keimey, 1982; Goldberg & Callahan, 
1991; Kemp & Kuman, 1990). This common solution has reversed the first intention of student evaluation 
which was to improve the quality of teaching (Rebell, 1 990). Student ratings also extend to course enjoyment 
and this can attract negative evaluations at all levels of student ability if the lecturer presents information 
at odds with their current values (Dershowitz, 1994; Stone, 1995). 

The Three Ability Framework (3AF) 

A complete alternative to postmortem student opinionnaires and their attendant problems is the Three 
Ability Framework (3AF) and its attendant advantages to the institution, faculty and students. The 3AF 
form has only 6 necessary ratings that take less than 5 minutes to complete. The form can be used many 
times by the lecturer during the course for in-course tracking of teaching quality. At the end of the course 
it can be used by the administration to give a single decision point number representing the quality of 
teaching. Previous research has identified teaching and assessment problems on professional courses with 
staff/student miss-matched expectations of three abilities. These three abilities are technical skills, 
professional competence and professional attitudes (Bastick, 1995). The 3AF uses the matching of staff/ 
student expectations on these three abilities as the bases of teaching effectiveness. 



The 3 AF form asks for two ratings of each of these abilities; ratings of how it is now on the course and 
ratings of how the student would want it to be. This is shown in figure 1 . The form takes longer to complete 
when additional information is requested. 
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Estimate, for you personally, how much this course emphasises, and should emphasise (i) Skills, 
(ii) Competence and (iii) Attitudes? Do this for both how the course is now, and for how the 
course should be - write a number in each box. 

As it is now on this course As it should be on this course 



(i) Emphasis on Skills 



Your 
estimate 
out of 100 



Your 
estimate 
out of 100 



As it is now on this course As it should be on this course 



(ii) Emphasis on Competence 



Your 
estimate 
out of 100 



Your 
estimate 
out of 100 



As it is now on this course As it should be on this course 



(iii) Emphasis on Attitudes 



Your 
estimate 
out of 100 



Your 
estimate 
out of 100 



Figure 1 : 3 AF form asking for two ratings of each of the three abilities 



Technical Skills refer to the traditional speed and accuracy of reproducing facts and processes and is 
assessed by timed accuracy of reproduction. Professional competence refers to the ability to use the skills 
in a novel situation or extend these skills in a novel way. The assessment is by justification of the 
appropriateness of what is done. Professional attitudes refers to values that are appropriate to the subject. 
They are assessed by demonstration in practical situations. 

From the difference in each pair of ratings it is possible to calculate the each student’s expectation for 
change in that ability. The lecturer completes the same ratings at the same time, and from the lecturer’s 
form it is possible to also calculate, in the same way, the lecturer’s expectations for change in each of the 
three abilities. Previous research has shown that when the students’ expectations are the same as the 
lecturer’s, that is both students and lecturer are working towards the same degree of change (indicator of 
effectiveness), then students get high grades and both the lecturer and the students enjoy the course (criteria 
of effectiveness). The correlations between the in-course indicators and the post-course criteria are 
significant at p<=0.002, with n=56 (Bastick, 1995). 

Measuring effectiveness of teaching (£T) 

The 3AF involves more than using the form and calculating the results. There are four steps in the 
application of the method 

1. Pre-course peer justification of ratings 

In the design stage, before the course starts, the lecturer needs to use his/her professional expertise to 
decide on what should be the emphasis on the course for each of the three abilities. A rationale should 
justify this decision, and it needs to be peer agreed. The lecturer can then build his/her expectations of the 
three abilities into the teaching and the design of course assignments that give students the opportunities to 
demonstrate the required level of each ability. ___ 
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2. Explain and justify three abilities to students 

Near the beginning of the course the lecturer needs to make sure the students understand the three 
abilities and how they will be taught and assessed through the content of the course. 

3. Monitor *as-is* and ‘should-be* for students and lecturer 

The 3AF form is completed by students and the lecturer when the lecturer is ready to monitor the 
course, or the administration is ready to assess the teaching. 

4. Calculating the effectiveness of teaching 

This is done in the following two stages (a) and (b). 

a) Calculate expected change for each student and lecturer: 

Change=(‘ should-be ’ as-i s ’ )/’as-is ’ 

b) Calculate alignment: 

Alignment=|Lecturer change-Student changej 

Zero is the perfect score 

The ET score can be calculated for each student or for any group of students. Hence, the effectiveness 
of teaching can be monitored for any group of interest - older students, students taking special electives, 
minority groups, gender balance, etc. Adjustments can be made to in-course teaching as necessary. 

Safeguards from variation in students’ set expectations 

The variation in students expectations can be calculated and used as a safeguard to protect the lecturer 
from inappropriate student expectations. It will be realized that all course evaluations depend on matching 
student expectations to the expectations of the course. Some aspects of the course can be presented in 
different ways to match different student expectations. For example, so called Teaming styles’ can be 
matched by adopting different ‘teaching styles’. However, some aspects of the course cannot be changed 
to match student expectations, e.g. the content or the peer agreed emphases of the three abilities. If student’s 
expectations of these unchangeables cannot be altered to accept them, then teaching ratings will go down 
through no fault of the lecturer. Traditional opinionnaires penalize the lecturer because they make no 
allowance for large variations in unchangeable student expectations. However, the 3AF allows the lecturer 
to show evidence that the original peer agreed emphases may not be appropriate for some groups on the 
course. This evidence can be used to either change the course expectations or change the student selection 
criteria. 



Institutional commitment to staff development 

The full Framework includes the commitment of the institution to develop faculty’s ability to use 
their subject specialism as a vehicle for explaining, teaching and assessing the three abilities. Institutional 
staff development support includes promoting academic freedom and professional responsibility, assisting 
faculty in making expectations explicit, in designing assessment opportunities for the three abilities and 
developing the ability of faculty to teach the three abilities using the content of their subject areas. There is 
a saying in business that ‘what gets rewarded gets done’. Quality teaching and quality learning are rewarded 
by the 3 AF. 
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